
United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Ofdce 
Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria. Virginia 223 1 3-1 450 
www.uspto.gov 



APPLICATION NO. 


FILING DATE 


FIRST NAMED INVENTOR 


ATTORNEY DOCKET NO. | 


CONFIRMATION NO. 


09/621,513 


07/21/2000 


James E. Mower 


0794.022 


4849 



23405 7590 08/16/2004 

HESLIN ROTHENBERG FARLEY & MESITI PC 
5 COLUMBIA CIRCLE 
ALBANY, NY 12203 



EXAMINER 



SOLOMON, GARY L 



ART UNIT 



PAPER NUMBER 



2615 

DATE MAILED: 08/16/2004 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO-90C (Rev. 10/03) 



Office Action Summary 


Application No. 

09/621,513 


Appllcant(s) 

MOWER. JAMES E. 


Examiner 

Gary L Solomon 


Art Unit 

2615 





" The MAILING DATE of this communication appears on the cover sheet with the correspondence address 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of tinne may be available under the provisions of 37 CFR 1 .136(a). In no event, hovvever, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

I) 13 Responsive to communication(s) filed on 30 June 2004 . 

2a)n This action is FINAL. 2b)13 This action is non-finaL 

3) n Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1 935 CD. 1 1 , 453 O.G. 21 3. 

Disposition of Claims 

4) 13 Claim(s) 1-54 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) n Claim(s) is/are allowed. 

6) KI Claim(s) 1-54 is/are rejected. 
?)□ Claim(s) is/are objected to. 

8) 0 Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) 0 The specification is objected to by the Examiner. 

10)0 The drawing(s) filed on is/are: a)D accepted or b)n objected to by the Examiner. 

Applicant may not request that any abjection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
Replacement drawing sheet(s) including the correction Is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

I I) 0 The oath or declaration is objected to by the Examiner. Note the attached Office Action or fonm PTO-152. 

Priority under 35 U.S.C. § 119 

1 2)0 Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 1 9(a)-(d) or (f). 
a)n All b)n Some * c)^ None of; 

1 0 Certified copies of the priority documents have been received. 

2.n Certified copies of the priority documents have been received in Application No. . 

30 Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attachment (s) 

1) □ Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-413) 

2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) Paper No(s)/Maii Date. . 

3) □ Information Disclosure Statement(s) (PTO-1449 or PTO/SB/08) 5) □ Notice of Informal Patent Application (PTO-152) 

Paper No(s)/Mail Date . 6) Q Other: . 



U.S. Patent and Trademark Office 

PTOL-326 (Rev. 1-04) 



Office Action Summary 



Part of Paper No./Mail Date 08082004 



Application/Control Number: 09/621,513 Pag 
Art Unit; 2615 

DETAILED ACTION 
Continued Examination Under 37 CFR LI 14 

1 . A request for continued examination under 37 CFR 1.114, including the fee set 
forth in 37 CFR 1 .17(e), was filed in this application after final rejection. Since this 
application is eligible for continued examination under 37 CFR 1.114, and the fee set 
forth in 37 CFR 1 .17(e) has been timely paid, the finality of the previous Office action 
has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 6-30- 
2004 has been entered. 

Response to Arguments 

2. Applicant's arguments filed 06-30-2004 have been fully considered but they are 
not persuasive. 

3. Applicant submits EUenby cannot disclose, teach or suggest registering the digital 
image with anything, let a digital image of the scene, on Page 14, Lines 5-6, 

The term, "digital elevation model", referred to in claim 1, is read by the 
examiner with the broadest reasonable interpretation. Merriam Webster defines model as 
'a miniature representation of something." In the present situation, the image displayed 
of the mountain in Figure 9 is a miniature representation of the mountain. 

The vision system present in EUenby operates in the digital domain (Column 2, 
Lines 22-24; Column 6, Lines 35-51). 

Therefore, the model in Figure 9 of the mountain clearly represents a digital 
elevation model. 

Second, the first and second images are registered with each other to form the 
composite image (Column 5, Lines 10-67). 
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Applicant submits a communications network is not required nor is disclosed in 
Ellenby et al. that is used for image exchange. 
Examiner disagrees. 

Figures 7 and 8 and Column 6, Line 35 through Column 7, Line 34 clearly 
illustrate how the system is configured to transfer an image from an officer to a helicopter 
and communicate to allow the officer to stay in pursuit of a fugitive. The 
communications network is maintained over radio transmission (Column 6, Line 55-60). 

Applicant submits Ellenby et al. cannot disclose or teach or suggest augmenting 
the registered digital image in any way, let alone the augmenting. 

Examiner disagrees. 

Ellenby does not explicitly teach the registering of the image, but it is inherent in 
his invention that some type of registration or tagging of the image data is required to 
augment the different images into one composite image. 

Column 3, Lines 26-37 and Figure 1 specifically teaches augmenting and 
registering multiple images together over a communications network. 

Claim Rejections - 35 USC §102 
4. The following is a quotation of the appropriate paragraphs of 35 U.S. C. 102 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 122(b), by 
another filed in the United States before the invention by the applicant for patent or (2) a patent granted 
on an application for patent by another filed in the United States before the invention by the applicant 
for patent, except that an international application filed under the treaty defined in section 351(a) shall 
have the effects for purposes of this subsection of an application filed in the United States only if the 
international application designated the United States and was published under Article 21(2) of such 
treaty in the English language. 



Application/Control Number: 09/621,513 Pag 
Art Unit: 2615 

5. Claims 1, 2, 5, 6, 7, 14, 15, 16, 17, 18, 20, 21, 24, 25,^-38, 41-42, and 49-54 
rejected under 35 U.S.C. 102(e) as being by being anticipated by Ellenby (US 6,370,556). 

For claim 1, Ellenby discloses a method of augmenting an image of a scene, 
comprising: 

obtaining by a first computing unit (Figure 1) a digital image of a scene; 
The vision system present in Ellenby operates in the digital domain (Column 2, 
Lines 22-24; Column 6, Lines 35-51). 

Therefore, the model in Figure 9 of the mountain clearly represents a digital elevation 
model. 

acquiring by the first computing unit a digital elevation model of the scene, 
wherein the digital elevation model comprises data for creating a three dimensional 
perspective model of the scene (Figure 9; Column 7, Lines 25-46); 

registering by the first computing unit the digital image with the digital elevation 
model to create a registered digital image (Figure 9); 

providing the registered digital image from the first computing unit to a second 
computing unit coupled to the first computing unit by a communications network 
(Column 3, Lines 14-46); and 

Figures 7 and 8 and Column 6, Line 35 through Column 7, Line 34 clearly 
illustrate how the system is configured to transfer an image from an officer to a helicopter 
and communicate to allow the officer to stay in pursuit of a fugitive. The 
communications network is maintained over radio transmission (Column 6, Line 55-60). 
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augmenting the registered digital image provided to the second computing unit 
with at least some of the information about the scene in response to input from the second 
computing unit ( Column 3, Lines 26-37). 

Column 3, Lines 26-37 and Figure 1 specifically teaches augmenting and 
registering multiple images together over a communications network. 

Ellenby does not explicitly teach the registering of the image, but it is inherent in 
his invention that some type of registration or tagging of the image data is required to 
augment the different images into one composite image. 

For claim 2, Ellenby discloses all the previous limitations, wherein obtaining the 
digital image comprises obtaining an image of the scene in real time with a fixed image 
capturing device (Column 2, Line 47). 

For claim 5, Ellenby discloses all the previous limitations, wherein the obtaining 
comprises obtaining an analog image and digitizing the analog image (Column 2, Lines 
21-24). 

For claim 6, Ellenby discloses all the previous limitations, wherein the acquiring 
comprises acquiring a preexisting digital elevation model (Figure 9). 

For claim 7, Ellenby discloses all the previous limitations, wherein the 
information comprises a location of at least one above-ground feature within the scene 
(Figure 11). 

For claim 14, Ellenby discloses all the previous limitations, further comprising 
rendering the digital image at the second computing unit to produce a rendered image, 
wherein the augmenting comprises: 
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identifying at the second computing a point of interest in the rendered image 
(Figure 9, Element 94); 

providing the identified point of interest from the second computing unit to the 
first computing unit (Figure 9, Element 94); 

reverse projecting the identified point of interest at the first computing unit to 
determine coordinates thereof (Column 7, Lines 25-60); and 

accessing the information at the first computing unit corresponding to the 
coordinates (Column 7, Lines 25-60). 

For claim 15, Ellenby discloses all the previous limitations, wherein the 
augmenting further comprises overlaying the information corresponding to the 
coordinates on the registered digital image (Figure 9). 

For claim 16, Ellenby discloses all the previous limitations, v^herein the 
information comprises textual cartographic data (figure 9). 

For claim 17, Ellenby discloses all the previous limitations, wherein the 
information comprises graphical cartographic data (Figure 9). 

For claim 18, Ellenby discloses all the previous limitations, farther comprising 
rendering the registered digital image at the second computing point unit to create a 
displayed image, wherein the input comprises pointing by a user to at least one area of 
the displayed image (Figure 9, Element 91). 

For claim 20, Ellenby discloses a method of augmenting an image of a scene, 
comprising: 

a computer unit programmable for: 

obtaining by a first computing unit (Figure 1) a digital image of a scene; 
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The vision system present in Ellenby operates in the digital domain (Column 2, 
Lines 22-24; Column 6, Lines 35-51). 

Therefore, the model in Figure 9 of the mountain clearly represents a digital elevation 
model. 

acquiring by the first computing unit a digital elevation model of the scene, 
wherein the digital elevation model comprises data for creating a three dimensional 
perspective model of the scene (Figure 9; Column 7, Lines 25-46); 

registering by the first computing unit the digital image with the digital elevation 
model to create a registered digital image (Figure 9); 

providing the registered digital image from the first computing unit to a second 
computing unit coupled to the first computing unit by a communications network 
(Column 3, Lines 14-46); and 

Figures 7 and 8 and Column 6, Line 35 through Column 7, Line 34 clearly 
illustrate how the system is configured to transfer an image from an officer to a helicopter 
and communicate to allow the officer to stay in pursuit of a fugitive. The 
communications network is maintained over radio transmission (Column 6, Line 55-60). 

augmenting the registered digital image provided to the second computing unit 
with at least some of the information about the scene in response to input from the second 
computing unit ( Column 3, Lines 26-37). 

Column 3, Lines 26-37 and Figure 1 specifically teaches augmenting and 
registering multiple images together over a communications network. 
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Ellenby does not explicitly teach the registering of the image, but it is inherent in 
his invention that some type of registration or tagging of the image data is required to 
augment the different images into one composite image. 

The unit, which augments the images together, must have been programmed by 
some computer code in order for it to take place. 

For claim 21, Ellenby discloses all the previous hmitations, wherein obtaining the 
digital image comprises obtaining an image of the scene in real time with a fixed image 
capturing device (Column 2, Line 47). 

For claim 24, Ellenby discloses all the previous limitations, wherein the obtaining 
comprises obtaining an analog image and digitizing the analog image (Column 2, Lines 
21-24). 

For claim 25, Ellenby discloses all the previous limitations, wherein the acquiring 
comprises acquiring a preexisting digital elevation model (Figure 9). 
For claim 32, refer to claim 14. 
For claim 33, refer to claim 15. 
For claim 34, refer to claim 16, 
For claim 35, refer to claim 17. 
For claim 36, refer to claim 19. 

For claims 37 -38, 41-42, and 49-53, refer to claims 1-2, 6-7, and 14-18. Image 
processing must be done by a computer program or other executable files. Program 
storage devices include floppy disks, CD ROMs, and hard drives etc. 

For claim 54, refer to previous rejections of claim 1, 20, and 37. 
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Claim Rejections - 35 USC § 103 
6. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

1. Claims 3, 9, 10, 12,^2, 27, 28, 30, 39,44,45, and 47 are rejected under 35 U.S.C. 
103(a) as being unpatentable over EUenby (US 6,307,556). 

For claim 3, Ellenby discloses all the previous Umitations, but fails to disclose 
wherein obtaining the digital image comprises obtaining an image of the scene in real 
time with a mobile image capturing device. However, it is notoriously well known that 
imaging devices can be mobile. Mobile imaging devices are able to capture images, 
which may be moving (Figure 6; The helicopter is flying and taking an image). 

Therefore, official notice is hereby given and thus it would have been obvious to 
one of ordinary skill in the art at the time of invention to configure the system of Ellenby 
with a mobile image-capturing device in order to capture images, which may be on the 
move (Figure 6; The helicopter is flying and taking an image). . 

For claims 9 and 10, Ellenby discloses all the previous limitations wherein the 
obtaining comprises obtaining an image of the scene from an image-capture device, the 
method further comprising determining an orientation and positional information for the 
image capture device (Fig. 10; Column 7, Line 62 through Column 8, Line 30). 

However, Ellenby doesn't distinctly show a zoom factor for the images, although 
it is implied in Figure 8. Zoom factors are well known in the art and have been used to 
place images in computer generated images and scenes. Therefore, it would have been 
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obvious to one of ordinary skill in the art at the time of the invention to include a zoom 
factor to determine an orientation of the image. 

For claim 12, Ellenby discloses all the previous limitations and also wherein the 
image capture device is fixed, and wherein the determining comprises reading positional 
information from the image-capturing device and obtaining the orientation from the 
equipment positionally fixed relative to the image capturing device. 

However, Ellenby doesn't distinctly show a zoom factor for the images, although 
it is implied in Figure 8. Zoom factors are well known in the art and have been used to 
place images in computer generated images and scenes. Therefore, it would have been 
obvious to one of ordinary skill in the art at the time of the invention to include a zoom 
factor to determine an orientation of the image. 

For claim 19, Ellenby discloses all the previous limitations, and also wherein the 
communication means exchange information from other vision systems. It would be 
obvious to one of just ordinary skill in the art at the instant of the invention to use a 
global communications network to allow for information exchange. Motivation would 
include transferring information globally. Official notice is hereby given that transferring 
information over a global communications network is well known. Ellenby teaches the 
exchange of information form different perspectives through communication (Column 2, 
Lines 55-68). Therefore it would have been obvious to one of ordinary skill in the art at 
the time of the invention to use a global information exchange. 

For claim 22, Ellenby discloses all the previous Hmitations, but fails to disclose 
wherein obtaining the digital image comprises obtaining an image of the scene in real 
time with a mobile image capturing device. However, it is notoriously well known that 
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imaging devices can be mobile. Mobile imaging devices are able to capture images, 
which may be moving (Figure 6; The helicopter is flying and taking an image). . 

Therefore, official notice is hereby given and thus it would have been obvious to 
one of ordinary skill in the art at the time of invention to configure the system of EUenby 
with a mobile image-capturing device in order to capture images, which may be on the 
move. 

For claims 27 and 28, Ellenby discloses all the previous limitations wherein the 
obtaining comprises obtaining an image of the scene from an image-capture device, the 
method further comprising determining an orientation and positional information for the 
image capture device (Fig. 10; Column 7, Line 62 through Column 8, Line 30). 

However, Ellenby doesn't distinctly show a zoom factor for the images, although 
it is implied in Figure 8. Zoom factors are well known in the art and have been used to 
place images in computer generated images and scenes. Therefore, it would have been 
obvious to one of ordinary skill in the art at the time of the invention to include a zoom 
factor to determine an orientation of the image. 

For claim 30, Ellenby discloses all the previous Umitations and also wherein the 
image capture device is fixed, and wherein the determining comprises reading positional 
information from the image-capturing device and obtaining the orientation from the 
equipment positionally fixed relative to the image capturing device. 

However, Ellenby doesn't distinctly show a zoom factor for the images, although 
it is implied in Figure 8. Zoom factors are well known in the art and have been used to 
place images in computer generated images and scenes. Therefore, it would have been 
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obvious to one of ordinary skill in the art at the time of the invention to include a zoom 
factor to determine an orientation of the image. 

For claims 39, 44, 45, and 47, refer to claims 3,9,10, and 12. A computer program 
or other executable files must do image processing. Program storage devices include 
floppy disks, CD ROMs, and hard drives etc. 

8. Claims 4, 8, 1 1, 13, 23, 26, 29, 31, 40, 43, 46, and 48 are rejected under 35 U.S.C. 
103(a) as being unpatentable over Ellenby (US 6,307,556) in view of Matsuoka (US 
2001/0028350). 

For claim 4, Ellenby discloses all the previous Umitations, but fails to disclose wherein 
obtaining the digital image comprises obtaining a preexisting image of the scene. 

However, Matsuoka discloses a maps database device in which a map is displayed 
based on coordinates of a location disclosed by a GPS sensor. The maps are displayed 
based on where the device is located. The augmented reality vision system synthesizes 
images based on a coordinate system. Configuring the two devices together would allow 
for the advantage of combining preexisting images and for images taken from an image 
capture unit in real time. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to combine the systems of Ellenby and Matsuoka in order to 
achieve an augmented reality vision system that synthesizes images in real time with a 
maps database that could synthesize both images together. 

For claim 8, Ellenby discloses all the previous limitations, and Matsuoka 
discloses wherein the information comprises a location of at least one underground 
feature within the scene (Figure 17). 
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For claim 1 1, EUenby and Matsuoka disclose all the previous limitations and also 
wherein the positional information comprises latitude, longitude (Figure 2; Matsuoka), 
and elevation above ground (Figure 9; EUenby). 

For claim 13, EUenby and Matsuoka disclose all the previous hmitations and also 
wherein the equipment comprises a global positioning system (Matsuoka; Figure 1, 
Element IC), a digital compass (Matsuoka; Figure 1 1) and a digital inclinometer 
(Matsuoka; Figure 13). 

For claim 23, EUenby discloses all the previous limitations, but fails to disclose 
wherein obtaining the digital image comprises obtaining a preexisting image of the scene. 

However, Matsuoka discloses a maps database device in which a map is displayed 
based on coordinates of a location disclosed by a GPS sensor. The maps are displayed 
based on where the device is located. The augmented reality vision system synthesizes 
images based on a coordinate system. Configuring the two devices together would allow 
for the advantage of combining preexisting images and for images taken from an image 
capture unit in real time. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to combine the systems of EUenby and Matsuoka in order to 
achieve an augmented reality vision system that synthesizes images in real time with a 
maps database that could synthesize both images together. 

For claim 26, EUenby discloses all the previous limitations, and Matsuoka 
discloses wherein the information comprises a location of at least one underground 
feature within the scene (Figure 17). 
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For claim 29, Ellenby and Matsuoka disclose all the previous limitations and also 
wherein the positional information comprises latitude, longitude (Figure 2; Matsuoka), 
and elevation above ground (Figure 9; Ellenby). 

For claim 3 1, Ellenby and Matsuoka disclose all the previous limitations and also 
wherein the equipment comprises a global positioning system (Matsuoka; Figure 1, 
Element IC), a digital compass (Matsuoka; Figure 1 1) and a digital inclinometer 
(Matsuoka; Figure 13). 

For claims 40, 43, 46, and 48, refer to claims 4, 8, 11, and 13. A computer 
program or other executable files must do image processing. Program storage devices 
include floppy disks, CD ROMs, and hard drives etc. 

Conclusion 

9. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Gary L Solomon whose telephone number is (703)-305- 
4370. The examiner can normally be reached on Monday - Friday 8:00 AM - 5:00 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
primary, Ngoc-Yen Vu can be reached on (703)-305-4946. The fax phone number for 
the organization where this application or proceeding is assigned is 703-872-9306. 
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Information regarding the status of an application may be obtained from the 
Patent AppHcation Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. Status 
information for unpublished applications is available through Private PAIR only. For 
more information about the PAIR system, see http://pair-direct.uspto.gov. Should you 
have questions on access to the Private PAIR system, contact the Electronic Business 
Center (EBC) at 866-217-9197 (toll-free). 
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